Quantitative, multispectral image analysis of tissue specimens stained with quantum dots

ABSTRACT

A biological sample such as a tissue section is stained with one or more quantum dots and possibly other fluorophores (total number of fluorophores N). A camera coupled to a microscope generates an image of the specimen at a plurality of different wavelengths within the emission spectral band of the N fluorophores. An analysis module calculates coefficients C 1  . . . C N  at each pixel from the set of images and reference spectral data for the N fluorophores. The coefficients C 1  . . . C N  are related to the concentration of each of the individual fluorophores at each pixel location. Morphological processing instructions find biological structures, e.g., cells, cellular components, genes, etc., in the images of the specimen. Quantitative analysis is performed on the identified biological structures. A display module displays the quantitative analysis results to the user, along with images of the specimen. The images can include images constructed from one or more of the coefficients C 1  . . . C N . The quantitative analysis display includes histograms of the biological structures, scatter plots of fluorophore concentrations, statistical data, spectral data and still others.

PRIORITY

This application claims priority benefits under 35 U.S.C. §119(e) toprior U.S. provisional application Ser. No. 60/876,493 filed Dec. 20,2006, the contents of which are incorporated by reference herein.

BACKGROUND

This invention relates to the field of systems and methods for analysisof biological specimens such as tissue sections, blood, cell culturesand the like. More particularly, this invention relates to a system,method and apparatus for analysis of images of biological specimenswhich are stained with one or more fluorophores, at least one of whichis a nano-crystalline luminescent semiconductor material known in theart as a “quantum dot.” This invention also relates to methods ofpresentation of quantitative data resulting from such analysis to auser.

It is known in the art that biological specimens, such as tissuesections from human subjects, can be treated with a stain containing anorganic fluorophore conjugated to an antibody which binds to protein,protein fragments, or other targets in the specimen. The stainedspecimen is then illuminated with light and the stain fluoresces. Adigital camera attached to a microscope is then used to capture an imageof the specimen. The areas where the fluorophore/antibody combinationbecame bound to the target of interest (e.g., proliferation proteinproduced by cancerous cells) appears as colored regions in the image ofthe specimen, with the color of the area dictated by the fluorescencespectrum of the fluorophore applied to the specimen. In addition to thevisible spectrum, the fluorescence signal may be detected in theinfra-red or ultra-violet regions, depending on emission spectrum of theparticular fluorophore. A stain containing two or more fluorophores canalso be applied to the specimen. These methods have a variety of uses,including diagnosis of disease, assessment of response to treatment, anddevelopment of new drugs to fight disease.

More recently, quantum dots have been developed as a stain material forbiological staining and imaging applications. The use of quantum dotsposes several advantages over traditional organic fluorophores for usein biological staining applications. These advantages include narrowemission band peaks, broad absorption spectra, intense signals, andstrong resistance to bleaching or other degradation.

Prior art references disclosing quantum dots and their application tobiochemical imaging applications include U.S. Pat. Nos. 6,322,901,5,990,749, and 6,274,323. Representative image capture and analysissystems and related methods are disclosed in the U.S. Pat. Nos.6,215,892 and 6,403,947 and published PCT applications WO 00/31534, WO00/17808 and WO 98/43042. Other prior art of interest includes US PatentApplication Publication US 2001/0033374 A1; US Patent ApplicationPublication 2002/0001080 A1; Fountaine et al., Multispectral imaging ofclinically relevant cellular targets in tonsil and lymphoid tissue usingsemiconductor quantum dots, Modern Pathology (2006) 1-11, and Huth etal., Fourier Transformed Spectral Bio-Imaging for Studying theIntracellular Fate of Liposomes, Cytometry Part A, vol. 57A pp. 10-21(2004). The entire content of the above-cited references areincorporated by reference herein.

SUMMARY

The following embodiments and aspects thereof are described andillustrated in conjunction with systems, tools and methods which aremeant to be exemplary and illustrative, not limiting in scope. Allquestions regarding scope of the invention are to be determined withreference to the appended claims and claims hereafter introduced intothe application.

In a first aspect, a system is disclosed for analysis of a biologicalspecimen. The specimen may, for example, take the form of a tissuesection obtained from a human or animal subject. The specimen may beliving cellular tissue, frozen cells, tumor cells, blood, throatculture, or other; the type or nature of specimen is not particularlyimportant. Typically, the specimen is mounted on a slide for analysis.The analysis may be for purposes of identification and study of thesample for presence of proliferation proteins or tumor cells, or forother purposes such as genomic DNA detection, messenger RNA detection,protein detection, or other. The biological specimen has between 1 and Ndiscrete fluorophore(s) applied to the specimen, the fluorophoresincluding at least one quantum dot. For example, the specimen may betreated with 2, 3 or 5 different quantum dots (N=2, 3 or 5 in theseexamples). One or more of the fluorophores applied to the specimen maybe organic fluorophores.

The system includes a microscope and attached digital camera capturingimages of the specimen. Each image is composed of a plurality of pixelscorresponding to the individual picture elements (pixels) of the digitalcamera. The camera captures an image of the specimen at a plurality ofdiscrete wavelengths. The number of wavelengths (M herein) may be 5, 10,20 or more. The wavelengths include discrete wavelengths at which the 1. . . N fluorophores produce a luminescent response to incident light. Adata set representing two dimensional pixel data at M wavelengths isreferred to herein occasionally as an “image cube.”

The system further includes a workstation which includes a processingunit executing software instructions which performing certain processingsteps on the images generated by the camera. These processing stepsinclude:

a) an unmixing process, which processes the plurality of images inconjunction with reference spectral data associated with the 1 . . . Nfluorophores and responsively calculates coefficients C₁ . . . C_(N) ateach pixel location, wherein the coefficients C₁ . . . C_(N) are relatedto the concentrations of the 1 . . . N fluorophores present in thesample at each pixel location;

b) at least one morphological processing process identifying at leastone biological structure in the specimen;

c) a quantitative analysis process calculating fluorophoreconcentrations for the biological structures identified by process b)from the coefficients C₁ . . . C_(N); and

d) a display process for displaying the results of the quantitativeanalysis process c) on a display associated with the workstation.

A variety of display tools are disclosed by which a user may interactwith the system and obtain displays of quantitative results from thespecimen. In one embodiment, the biological structures which areidentified by the morphological processing process are cells or cellularcomponents. The morphological processing process measures the size ofthe biological structures, and counts the number of biologicalstructures identified in the specimen. The results of the quantitativeanalysis process are presented as a histogram of the number ofbiological structures sorted by size of the biological structures. Thehistograms may also include histograms of the size distribution of cellshaving a positive signal for each of the 1 . . . N fluorophores appliedto the specimen.

In another embodiment, the display process includes a feature allowing auser to select a segment of an image of the specimen displayed on thedisplay (e.g., a region of the sample having a high concentration ofcells with a high fluorescent signal) and the display process displaysquantitative results for the selected segment of the image. As a furtherenhancement, the quantitative results are displayed as a plot ofconcentration of one quantum dot as a function of concentration of asecond quantum dot for cells positive for both quantum dots. Such a plotcan visually be represented as a scatter plot. Scatter plots can bedisplayed for either the entire image or any selected sub-segment of theslide.

In another embodiment, the coefficients C₁ . . . C_(N) are scaled toabsolute concentrations of the fluorophores in the specimen (e.g.,nanomols per liter, number of quantum dots per cell, or other system ofunits). Furthermore, the plots of concentration of fluorophores can beexpressed in units of absolute concentrations.

The display of the quantitative results may include display of an imageof the specimen on the same display. The image can be constructed fromone or more of the M images, or, more preferably, from one or more ofthe coefficients C₁ . . . C_(N). It will be recalled that thecoefficients are obtained by the unmixing processes and are known foreach pixel. For example, if the user is studying a histogram of the sizedistribution of cells have positive signal for a quantum dot fluorophorewhose emission spectrum peaks at 625 nm, the display may simultaneouslyshow an image of the specimen with the image generated from thecoefficient C_(i) which corresponds to the 625 nm quantum dotfluorophore. In other words, the image masks (omits) the signalcontribution from all other fluorophores which may be present in thesample and only reveals the signal from the 625 nm fluorophore. Thequantitative results may further include statistical data for thesegment of the image selected by the user.

In one embodiment the display process further provides a tool by whichcolor intensity for one or more selected fluorophores can be selectivelyweighted by the user to thereby change the appearance of the image onthe display. For example, the user may wish to view an image of thespecimen that reveals the distribution of 605 nm and 625 nm quantum dotsin the specimen. When such image is displayed, a tool is presented bywhich a user can selectively weight (or attenuate), either the 605 nmquantum dot signal or the 625 nm quantum dot signal. The weighting maybe used for example to strengthen a weak fluorophore signal and allowthe user to more readily perceive the distribution of the fluorophore inthe biological structures (e.g., cells) in the specimen.

The display process may combine the various analytical features andprovide a variety of different tools for analyzing the specimen. Forexample, the display process may include processes for displaying i) animage of the specimen constructed from one or more of the coefficientsC₁ . . . C_(N) (either an image of the entire specimen or somesub-segment of the specimen); ii) a histogram of biological structuresidentified in the image in i) sorted by size of the biologicalstructures, for at least at least one of the fluorophores applied to thesample; and iii) one or more scatter plots of concentration of one ofthe fluorophores as a function of concentration of one of the otherfluorophores, for biological structures having a positive signal forboth fluorophores. These features may be combined with the display ofadditional statistical data, tools for selection of portions of an imageconducting further quantitative analysis, and still other features.

In still another embodiment, the display process includes a feature bywhich a user may select a portion of a scatter plot, histogram, or othervisualization of the quantitative data and conduct further quantitativeanalysis on the portions of the specimen corresponding to the selectedportion of the scatter plot, histogram or other visualization. Forexample, a user may select the portion of a histogram corresponding tolarger cells with relatively high concentrations of a particularfluorophore (e.g. 625 nm quantum dot). The display process creates a newdisplay which displays additional quantitative data for the larger cellsin the histogram which were selected by the user. Such quantitative datamay take a variety of forms, such has a new scatter plot showing theconcentration of the 605 nm quantum dot as a function of theconcentration of the 625 nm quantum dot, for the cells which correspondto the portion of the histogram selected by the user.

Additionally, the display process may display an image of the specimenwith the biological structures associated with the selected portion ofthe histogram, scatter plot or other visualization, with the biologicalstructures highlighted, e.g. in a contrasting color. The image can beconstructed from the concentration coefficient corresponding to the 625nm quantum dot, the 605 quantum dot, other fluorophore present in thesample, e.g., autofluorescence, combination thereof, or other.

In yet another aspect of this disclosure, a method is provided foranalysis of a biological specimen in which between 1 . . . N quantumdots are applied to the specimen. The method includes the steps of:

(a) capturing a set of images of the specimen with a camera coupled to amicroscope at M different wavelengths, where M is an integer greaterthan 2, the images arranged as an array of pixels;

(b) determining, from the set of M images, coefficients C₁ . . . C_(N)for each pixel, wherein the coefficients C₁ . . . C_(N) are related tothe concentrations of the 1 . . . N quantum dots present in the specimenimaged by each pixel;

(c) morphologically processing an image constructed from one or more ofthe coefficients C₁ . . . C_(N) to identify cells or cellular componentsin the specimen,

(d) conducting a quantitative analysis of cells or cellular componentsidentified in step (c) from the coefficients C₁ . . . C_(N); and

(e) displaying the results of the quantitative analysis process (d) on adisplay of a workstation.

The quantitative analysis and displaying steps may incorporate one ormore of the quantitative analysis and display features highlighted abovein the discussion of the system aspect of this invention.

In still another aspect of this disclosure, the invention can becharacterized as biological specimen analysis apparatus taking the formof a machine readable storage medium (e.g., hard disk, CD, or othermedium) which contains a set of software instructions for execution by aprocessing unit, e.g., a computer workstation. The processing unit hasaccess to an image cube of a specimen stained with one or more quantumdots and imaged with a camera coupled to a microscope. The image cubemay be a set of M images of the sample taken at M different wavelengths,where M is an integer greater than 2. The images are arranged as anarray of pixels. The set of images can be stored locally on theprocessing unit, obtained over a network, or also stored on themachine-readable storage medium. The instructions comprise a set ofinstructions for:

(a) determining from the set of M images coefficients C₁ . . . C_(N) foreach pixel, wherein the coefficients C₁ . . . C_(N) are related to theconcentrations of the one or more quantum dots present in the specimenimaged by each pixel;

(b) morphologically processing an image of the specimen to identifycells or cellular components in the specimen,

(c) conducting a quantitative analysis of the specimen includingcalculating quantum dot concentrations for the cells or cellularcomponents identified in step (b) from the coefficients C₁ . . . C_(N);and

(d) generating data for display of the results of the quantitativeanalysis process (c) on a display associated with the processing unit.

As with the method aspect of the invention, the software instructionsmay incorporate one or more of the quantitative analysis and displayfeatures highlighted above in the discussion of the system aspect ofthis invention.

The image analysis, quantitative analysis and display methods arepreferably designed to be used with a variety of commercially availableimaging platforms, staining systems and workstations. Accordingly, inone possible embodiment the software instructions can be provided as aseparate product that enables existing imaging equipment and computerworkstations to practice the invention, without necessitating purchaseof expensive new hardware. Thus, the software instructions stored on amachine readable medium (e.g., CD) have their own special utility andadvantage.

In addition to the exemplary aspects and embodiments described above,further aspects and embodiments will become apparent by reference to thedrawings and by study of the following detailed descriptions.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is block diagram of a system for analyzing a biological specimen.The system includes a digital camera, a microscope, and a workstationhaving a display and a memory storing software instructions forprocessing images of the specimen captured by the camera.

FIG. 2 is a more detailed block diagram of the workstation of FIG. 1,showing software modules which are stored in memory of the workstation.

FIG. 3 is a flow chart showing a sequence of processing steps performedby the initialization and set-up module of FIG. 2.

FIG. 4 is a flow chart showing a sequence of processing steps performedby the analysis module of FIG. 2.

FIG. 5 is an illustration of a set of images (M=22) of a slidecontaining a biological specimen, each of the images captured by thecamera of FIG. 1 at a discrete wavelength; data from the set of imagesof FIG. 5 are referred to herein occasionally as an “image cube.”

FIG. 6 is a graph of the reference emission spectra of seven quantum dotfluorophores, one or more of which are applied to the specimen. Thespectra of FIG. 6 are stored in the workstation and applied to the imagecube of FIG. 5 in a spectral unmixing algorithm in order to calculatecoefficients C₁ . . . C_(N) at each pixel location. The coefficients C₁. . . C_(N) are related to the concentrations of the 1 . . . Nfluorophores (quantum dots) present in the sample at each pixellocation.

FIG. 7 is a typical image of the biological specimen in which theluminescent emission from multiple quantum dots contribute to the imageand aid in highlighting cells or other cellular structures such asnuclei or cell membranes. The image of FIG. 7 can be generated from oneor more the coefficients C₁ . . . C_(N).

FIG. 8 is an illustration of a display presented on the workstationshowing the user selecting the fluorophores which are present in theimage, assigning an informative label to that fluorophore, and selectinga color to use to represent the individual fluorophores.

FIG. 9 is another image of the biological specimen, composed of two ofthe coefficients C₁ . . . C_(N).

FIG. 10 is an illustration of a display presented on the workstationshowing a feature by which a user can weight (i.e., enhance orattenuate) the contribution of one fluorophore or another in thegeneration of the image of FIG. 9. Note from FIG. 10 that one of thefluorophores may consist of autofluorescence from the specimen.

FIG. 11 is an illustration of a display on the workstation showing a setof quantitative data which is presented to the user for the specimen,including histograms, scatter plots, statistical data and image data.

FIG. 12 is an illustration of a display on the workstation showing twoseparate images of the specimen, one from autofluorescence and anotherfrom a quantum dot emitting at 625 nm, and spectral data for a selectedpoint or region in the images.

FIG. 13 is an illustration of two histograms showing quantitative dataand showing a feature by which a user can select a sub-population ofcells in one of the histograms and perform additional quantitative andqualitative analysis on the selected sub-population of cells.

FIG. 14 is an illustration of a display of the workstation showing ascatter plot providing additional quantitative analysis of the selectedsub-population of cells selected by the user in FIG. 13.

FIG. 15 is an illustration of a display presented on the workstationshowing an image of the specimen and a feature by which a user canselect a discrete sub-region of the image and have additionalquantitative analysis performed on the selected sub-region.

FIG. 16 is an illustration of a display presented on the workstationshowing two images of the same portion of the specimen, one showing theautofluorescence and the other the luminance from a 625 nm quantum dot,and spectra for the region of the specimen selected in the procedureshown in FIG. 15.

FIG. 17 is an illustration of a display presented on the workstationshowing three images of the same region of the specimen, one imageshowing the autofluorescence signal from the specimen, one showing thesignal from a 605 nm quantum dot, and one showing the signal from a 655nm quantum dot. FIG. 17 also shows the spectra for the region of thespecimen represented in the three images.

FIG. 18 is another illustration of a display presented on theworkstation showing quantitative data from the specimen in the form ofhistograms, scatter plots, statistical data, and image data.

DETAILED DESCRIPTION

System and Software Overview

FIG. 1 is a block diagram of a system for analysis of a biologicalspecimen 10. The specimen 10 may, for example, take the form of a tissuesection obtained from a human or animal subject, such as aformalin-fixed, paraffin-embedded tissue sample. The specimen may beliving cellular tissue, frozen cells, tumor cells, blood, throatculture, or other; the type or nature of specimen is not particularlyimportant.

Typically, the specimen is mounted on a slide 18 or other device forpurposes of imaging by a camera system platform 22. Computer analysis ofimages of the specimen is performed in a workstation 34 in accordancewith the present disclosure. The analysis may be for purposes ofidentification and study of the sample for presence of proteins, proteinfragments or other markers indicative of cancer or other disease, or forother purposes such as genomic DNA detection, messenger RNA detection,protein detection, detection of viruses, detection of genes, or other.

The biological specimen 10 is stained by means of application of a staincontaining one or more different fluorophore(s). The number N offluorophores that are applied to the specimen can vary, but willtypically be between 2 and say 10. The fluorophores may comprise one ormore nano-crystalline semiconductor fluorophores (i.e., quantum dots)12, each producing a peak luminescent response in a different range ofwavelengths. Quantum dots are described in the patent and technicalliterature, see for example U.S. Pat. Nos. 6,322,901, 5,990,749, and6,274,323. The term “quantum dot” is intended to be broadly read toencompass such structures generally. Quantum dots, including conjugatedquantum dots, are commercially available from Invitrogen Corp., EvidentTechnologies, and others.

For example, the specimen 10 may be treated with 2, 3 or 5 differentquantum dots (N=2, 3 or 5 in this example), for example quantum dotswhich produce a peak luminescent response at 525, 600 and 625 nm. One ormore of the fluorophores applied to the specimen may be organicfluorophores 14 (e.g., DAPI, Texas Red), which are well known in theart. Thus, the system of FIG. 1 can be used with a specimen which isstained with just quantum dots, with quantum dots in combination withconventional organic fluorophores, or just conventional organicfluorophores. It is noted that quantum dots have several importantadvantages over conventional organic fluorophores. In practice, thequantum dots or other fluorophores are conjugated to an antibody, whichis designed to bind to a target in the specimen, such as a protein.

In typical practice, the specimen is processed in an automatedstaining/assay platform 16 which applies a stain containing quantum dotsand/or organic fluorophores to the specimen. There are a variety ofcommercial products on the market suitable for use as the staining/assayplatform, one example being the Discovery™ product of the assigneeVentana Medical Systems, Inc.

After preliminary tissue processing and staining in the platform 16, theslide 18 containing the specimen 10 is supplied to a camera systemplatform 22. The platform 22 includes a light source for illuminatingthe specimen 10 at wavelengths intended to produce a luminescentresponse from the fluorophores applied to the specimen. In the case ofquantum dots, the light source may be a broad spectrum light source.Alternatively, the light source may comprise a narrow band light sourcesuch as a laser. The camera platform also includes a microscope 24having one or more objective lenses and a digital imager (camera) 26which is coupled to the microscope in order to record high resolution,magnified digital images of the specimen. As will be explained below,the specimen 10 is imaged by the camera 26 at a plurality of differentwavelengths. In order to capture images at a plurality of differentwavelengths, the camera platform 22 includes a set of spectral filters28. Other techniques for capturing images at different wavelengths maybe used. The camera 26 may take the form of a charge-coupled deviceimager sensitive to light in a band covering the luminescent responsespectra of the fluorophores, e.g., between 400 and 900 nm. Cameraplatforms suitable for imaging stained biological specimens are known inthe art and commercially available from companies such as Zeiss, Canon,Applied Spectral Imaging, and others, and such platforms are readilyadaptable for use in the system, methods and apparatus of thisinvention.

The camera 26 images the specimen at a plurality (M) of discretewavelengths and responsively generates an image of the specimen at eachof the M wavelengths. Each of the images is composed of a plurality ofpixels corresponding to the individual picture elements (pixels) in thedigital imager 26. The wavelengths at which the specimen is imagedincludes wavelengths at which the 1 . . . N fluorophores present in thesample produce a luminescent response to incident light. For example,suppose the specimen is stained with two quantum dots, having a peakluminescent response at 625 nm and 605 nm. Suppose further that thenominal (reference) spectra of such quantum dots has a Gaussiandistribution with appreciable response between 575 and 750 nm (see forexample the reference quantum dot spectra in FIG. 6 and the discussionbelow). Therefore, the camera 26 is operated to image the specimen atsay 10, 15, or 20 different wavelengths between 575 and 750 nm. As anexample, the camera (and any attendant spectral filters 28) is operatedso as to capture images of the specimen at 575, 600, 625, 650, 675, 700,725 and 750 nm, such wavelengths overlapping the reference spectra ofthe 605 and 625 nm quantum dot fluorophores at wavelengths where thefluorophores produce a significant luminescent response. M=8 in thisexample.

The data resulting from a set of M images of the specimen (one taken ateach of the M wavelengths) is referred to herein as an “image cube”.Referring again to FIG. 1, the image cube is supplied to the workstation34, either via a cable connection between the camera imaging platform 22and the workstation 34 (indicated at 46) or via a computer network 30connecting the camera imaging platform 22 to the workstation 34 or usingany other medium that is commonly used to transfer digital informationbetween computers. The image cube can also be supplied over the network30 to a network server 32 or database for storage and later retrieval bythe workstation 34.

The workstation 34 includes a central processing unit 36 and a memory36, user input devices in the form of a keyboard 40 and mouse 42, and adisplay 44. As will be explained in the following discussion, theprocessor 36 executes program instructions loaded in to the memory 38which perform analysis of the image cube, morphological processing ofthe images or image data derived from such images, quantitativeanalysis, and display of quantitative results to a user operating theworkstation 34.

FIG. 2 is a more detailed illustration of the workstation of FIG. 1showing in greater detail certain software modules and data stored inthe memory 38. The memory includes three main software instructionmodules, namely an initialization and set-up module 50 (steps of whichare shown in FIG. 3), an analysis module 52 (steps of which are shown inFIG. 4) and a display module 54, the operation of which will bedescribed in conjunction with FIGS. 7-18. The memory 38 further storescertain other data which is used or operated on by the modules 50, 52and 54, namely run-time libraries 60, configuration files 62, andreference spectral data 64 for the 1 . . . N fluorophores which areapplied to the sample. An example of the reference spectra 64 for anassay using seven different quantum dots (Qdot 1 . . . Qdot 7) is shownin FIG. 6. The reference spectra 64 are used in the spectral unmixingalgorithm in the analysis module 52, as will be explained in greaterdetail below. The spectra 64 of FIG. 6 are offered by way of exampleonly to show that different quantum dots have different spectra,including a different wavelength of peak luminescent response anddifferent peak intensity under the same illumination conditions. Thespectral data 64 will thus vary depending on the particular quantum dotsthat are used in an assay. The spectral data of FIG. 6 can be obtainedfrom the manufacturer of the quantum dots or alternatively obtained bytesting examples of the quantum dots using appropriate equipment.

The memory 38 further stores the image cube 70 comprising the M imagesof the spectrum, taken at M different wavelengths. The memory furtherstores a list 72 of the wavelengths at which the specimen was imaged,and a list 74 of the exposure times at each of the wavelengths. Theimage cube and lists 72 and 74 are inputs to the analysis module 52.

The memory further stores the calculated concentration coefficients C₁ .. . C_(N) (item 80). The coefficients are outputs from the analysismodule 52 and are used by quantitative analysis routines in the module52. The module 52 further produces as additional output quantitativedata 82 which is stored in memory. The concentration coefficients 80 andquantitative data 82 are used by the display module 54 to display thequantitative data to the user in a convenient and user-friendly fashionas will be explained below.

One aspect of the processing performed by the analysis module 52 in theworkstation 34 is a spectral unmixing process by which the plurality ofimages in the image cube 70 are processed with reference spectral data64 associated with the 1 . . . N fluorophores (FIG. 6) in order toproduce an estimate of coefficients C₁ . . . C_(N) at each pixellocation. The coefficients C₁ . . . C_(N) are related to theconcentrations of the 1 . . . N fluorophores present in the sample ateach pixel location. The coefficients C₁ . . . C_(N) can be scaled toabsolute fluorophore concentrations. The coefficients also can be scaledin arbitrary units and used to represent concentration in terms ofillumination intensity, either relative or absolute.

The term “pixel location” in the context of the coefficients C₁ . . .C_(N) will be understood to refer to the individual locations in each ofthe images which also corresponds to the individual pixels of thedigital camera 26. For example, if the digital camera 26 is constructedas an array of pixels arranged in 1 . . . i rows and 1 . . . j columnsof pixels, each pixel of the camera imaged the specimen M times.

For each pixel, the spectral unmixing process calculates coefficients C₁. . . C_(N) which relate to the concentration of each fluorophore to allof the M images for that pixel. When applied to the entire image cube,the spectral unmixing process determines, in an overall sense, therelative contributions of each of the 1 . . . N fluorophores present inthe sample to the resulting images, and in particular theirconcentrations, either in relative terms or with appropriate scaling inabsolute terms. The spectral unmixing process performs such calculationsfor each of the pixels. A variety of unmixing processes can be used, anda linear spectral unmixing process as described in Huth et al., FourierTransformed Spectral Bio-Imaging for Studying the Intracellular Fate ofLiposomes, Cytometry Part A, vol. 57A pp. 10-21 (2004) is consideredpreferred. This process will be discussed in greater detail below.

The workstation further includes software instructions as part of theanalysis module 52 which performs at least one morphological process inorder to identifying one or more biological structures in the specimen.Such structures can be whole cells (indicated at 20 in FIG. 1), orcellular components such as cell membranes, nuclei, cytosol,mitochondria, genes, DNA fragments, RNA, messenger RNA entities, orother, and are identified by shape or other characteristics which can bedetermined by using known morphological or similar image processingtechniques. The morphological processing to identify such structures canbe performed on any one of the images, all of the images, or morepreferably an image constructed from one or more of the identifiedcoefficients C₁ . . . C_(N). In still another possible variation, thespecimen is stained with Hematoxylin and Eosin (H and E) and themorphological process identifies the biological structures of interestfrom an image of the sample stained with H and E.

A variety of morphological processing techniques are known to personsskilled in the art which can be used to identify the biologicalstructures in an image of the sample. Examples include a multi-scaleapproach, such as described in Kriete, A et al., Automatedquantification of quantum-dot-labeled epidermal growth factor receptorinternalization via multiscale image segmentation, Journal ofMicroscopy, v. 222(1) 22-27 (April 2006); an active contour (snake)approach, described in Kass, A. Witkin, and D. Terzopoulos. Snakes:Active contour models. International, Journal of Computer Vision,1:321-332, 1988; a level set approach, described in J. A. Sethian, LevelSet Methods: Evolving Interfaces in Geometry, Fluid Mechanics, ComputerVision and Materials Sciences. Cambridge Univ. Press, 1996; a contourclosure approach described in Mahamud, S et al., Segmentation ofMultiple Salient Closed Contours from Real Images, IEEE Transactions OnPattern Analysis And Machine Intelligence, Vol. 25, No. 4, April 2003,and a Watershed approach (currently used in the illustrated embodiment),described in Vincent, L. et al., Watersheds in digital spaces: Anefficient algorithm based on immersion simulations, IEEE Transactions onPattern Analysis and Machine Intelligence v. 13(6) June 1991 pp.583-598, see also the review article on Watershed: Roerdink, J. andMeijster A., The Watershed Transform: Definitions, Algorithms andParallelization Strategies”, Fundamenta Informatica v. 41, 2001, IOSPress pp. 187-228. Other techniques can be used, including thosedisclosed in the following papers: Thouis R. Jones et al., Voroni-BasedSegmentation of Cells on Image Manifolds, in CVBIA, ser. Lecture Notesin Computer Science, Y. Liu et al. Eds., vol. 3765 Springer-Verlag, 2005pp. 535-543; the poster paper of Thouis R. Jones et al., Methods forHigh-Content, High-Throughput Image-Based Cell Screening, Proceedings ofMIAAB 2006 available on-line atwww.broad.mit.edu/˜thouis/MIAABPoster.pdf; and Gang Lin et al., A Hybrid3 D Watershed Algorithm Incorporating Gradient Cues and Object Modelsfor Automatic Segmentation aof Nuclei in Confocal Image Stacks,Cytometry Part A, 56A:23-26 (2003).

Once the biological structures are identified in the specimen usingthese processes, a routine in the analysis module 52 counts thestructures in the entire specimen, counts the structures positive foreach of the fluorophores applied to the specimen, measures their size,and stores the location in the image of each of such structures. Thestorage of such quantitative data is represented in FIG. 2 at 82.

Part of the analysis module 52 includes a quantitative analysis processwhich, among other things, calculates the fluorophore concentrations forthe biological structures 20 (FIG. 1) for each of the fluorophores,using the coefficients C₁ . . . C_(n) which were obtained in thespectral unmixing process. For example, for each of the identifiedbiological structures 20 (e.g., cells), the quantitative analysis sumsthe total fluorophore concentrations for each of the N fluorophores forthose pixels representing the biological structures. Consider, forexample, a specimen containing cancer cells stained with six quantumdots conjugated to antibodies which are designed to attach to sixdifferent proliferation proteins which may be found in the specimen.Consider further that two of the quantum dots (605 nm and 625 nm) werebound to the cells in the specimen but the remaining four quantum dotswere not bound to any cells in the specimen. The morphologicalprocessing processes identify all cells in the specimen which produced anon-zero luminescent response for the 605 and 625 quantum dots. (Athreshold other than zero could of course be specified, such as 10 or 30on a scale of 0-255 with 8-bit quantization of the signal level from thecamera 26.) The pixels coordinates for such cells are identified. Thevalues of coefficients C_(i), C_(j) associated with the 605 and 625quantum dots are recorded for such pixel coordinates, and optionallyscaled to intensities, absolute concentrations, or other value. Theresulting quantitative data, along with pixel addresses for the cells,is stored in memory in the workstation.

The quantitative analysis module may also calculate other statistics forthe sample, including (a) counts of the number of cells; (b) counts ofthe number of cells with positive signal for each of the fluorophores;c) sorting the cells into histograms organized by size, presence of oneor more fluorophores or other characteristic; (d) calculating mean,median and standard deviation of cell sizes; (d) measurements offluorophore concentration (intensity) for the identified biologicalstructures, and still others. Such additional quantitative data is alsorepresented in FIG. 2 at 82.

The workstation further includes a display process or module 54 fordisplaying the results of the quantitative analysis process c) on adisplay 44 associated with the workstation. A variety of methods andtools for display of quantitative data from the specimen arecontemplated and will be described with reference to FIG. 11-18 in thefollowing discussion. As one example, as shown in FIG. 11, the resultsof the quantitative analysis process are presented as a histogram of thenumber of biological structures sorted by size of the biologicalstructures, for each of the 1 . . . N fluorophores applied to thespecimen.

In another embodiment described later in conjunction with FIGS. 15 and16, the display process includes a feature allowing a user to select asegment of an image of the specimen displayed on the display (e.g., aregion of cells having a particularly high fluorescent signal) and thedisplay process displays quantitative results for the selected segmentof the image. As another example, as shown in FIGS. 11 and 14, thequantitative results that are displayed can take the form of a plot ofconcentration of one fluorophore (e.g. 605 quantum dot) as a function ofconcentration of a second fluorophore (e.g., 625 quantum dot) for thosecells positive for both fluorophores, either in a selected segment ofthe image, or in an overall image. Such a plot can visually berepresented as a scatter plot, for example as shown in FIG. 18.

In another embodiment, the coefficients C₁ . . . C_(N) are scaled toabsolute concentrations of the fluorophores in the specimen (e.g.,nanomols per cubic micron, number of quantum dots per cell, or othersystem of units). Furthermore, the plots of concentration offluorophores, or the histograms, or other reports or formats ofquantitative data for the specimen, can be expressed in terms ofabsolute concentrations of fluorophores present in the specimen, orselected portions of the specimen.

As shown for example in FIGS. 11 and 18, the display of the quantitativeresults may include display of an image of the specimen on the samedisplay. The image can be constructed from one or more of the M images.More preferably, the image that is displayed simultaneous with thequantitative data is constructed from one or more of the coefficients C₁. . . C_(N), it being recalled that such coefficients are obtained inthe unmixing processes and are determined for each pixel. For example,if the user is studying a histogram of the size distribution of cellshave positive signal for a 625 nm quantum dot fluorophore, the displaymay simultaneously show an image of the specimen with the imagegenerated from the coefficient C_(i) which corresponds to the 625 nmquantum dot fluorophore. Such image may be of the entire specimen, or aselected portion of the specimen. In other words, the image masks(omits) the luminescent response from all the other fluorophores whichmay be present in the specimen, and only reveals the contribution of the625 nm fluorophore. The quantitative results may further includestatistical data for the segment of the image selected by the user.

In a still further example, the display process provides a tool by whichcolor intensity for one or more selected fluorophores can be selectivelyweighted by the user to thereby change the appearance of the image onthe display. For example, the user may wish to view an image of thespecimen that reveals the distribution of cells positive for the 605 nmand 625 nm quantum dots in the specimen. When such image is displayed, atool is presented by which a user can selectively weight (or attenuate),either the 605 nm quantum dot signal or the 625 nm quantum dot signal.The weighting may be used for example to strengthen a weak fluorophoresignal and allow the user to more readily perceive the distribution ofthe fluorophore in the biological structures (e.g., cells) in thespecimen.

The display process may combine the various analytical features andprovide a variety of different tools for analyzing the specimen. Forexample, the display process may includes processes for displaying i) animage of the specimen constructed from one or more of the coefficientsC₁ . . . C_(N) for each pixel; ii) a histogram of biological structuresidentified in the image in i) sorted by size of the biologicalstructures, for at least at least one of the fluorophores applied to thesample; and iii) one or more scatter plots of concentration of one ofthe fluorophores as a function of concentration of one of the otherfluorophores, for cells or other biological structures positive for bothfluorophores. An example of such as display is shown in FIGS. 11 and 18.These features may be combined with the display of additionalstatistical data, tools for selection of portions of an image conductingfurther quantitative analysis, and still other features.

In still another embodiment, the display process further includes afeature by which a user may select a portion of a scatter plot or ahistogram, e.g., by drawing a box around the portion of the histogram orscatter plot using a mouse, and conduct further quantitative analysis onthe portions of the specimen (e.g., particular cells) corresponding tothe selected portion of the scatter plot or histogram. For example, auser may select the portion of a histogram corresponding to larger cellswith relatively high concentrations of a particular fluorophore (e.g.625 nm quantum dot). The display process creates a new display whichdisplays additional quantitative data for the larger cells in thehistogram which were selected by the user. Such quantitative data maytake a variety of forms, such has a new scatter plot showing theconcentration of the 605 nm quantum dot as a function of theconcentration of the 625 nm quantum dot, for the cells which correspondto the portion of the histogram selected by the user. Additionally, thedisplay process may display an image of the specimen with the biologicalstructures associated with just the cells in the selected portion of thehistogram presented in the image and highlighted, e.g., in a contrastingcolor. Such image could be constructed from the concentrationcoefficient C_(i) corresponding to the 625 nm quantum dot, the 605quantum dot, other fluorophore present in the sample, autofluorescence,combination thereof, or other.

The above-described software processes will now be described in greaterdetail with reference to FIGS. 3-18. As an initial step, the specimen isprocessed and stained with one or more fluorophores (e.g., up to Nquantum dots, where N may for example be 2, 5, 10 or more), and thenimaged at the M wavelengths as described above. The resulting imagecube, list of wavelengths and exposure times at each wavelength arestored in the memory 38 of the workstation 34.

FIG. 5 is an illustration of an example of one image cube 70, consistingof twenty-two discrete images 150, 152, 154 . . . of the specimen atdifferent wavelengths (M=22). The data representing the image cube 70 isobtained by the camera 26 of FIG. 1. The wavelengths λ₁ . . . λ_(M) areselected so that they overlap the portions of the reference spectra forthe fluorophore(s) applied to the sample where there is a significantluminescent response from the fluorophore(s). For example, image 150 isan image at 505 nm, image 152 is an image of the specimen at 515 nm,image 154 is an image of the specimen at 525 nm, and the remaining 19images obtained at 10 nm increments up to 715 nm. The 22 wavelengthsoverlap substantially the reference spectra for Qdot 1 . . . Qdot 5 ofthe reference spectra of FIG. 6.

The camera 26 (FIG. 1) can use any convenient filtering technique toacquire the spectral images forming the image cube, including the use ofphysical filters 28, e.g., Liquid Crystal Display (LCD) spectralfilters, or other types of filters.

Each image is captured for an appropriate exposure interval, dependingon the sensitivity of the camera and the intensity of the illuminationsource. Such exposure interval may for example be from between 100milliseconds and 5 seconds per sampled wavelength. The exposure intervalat each wavelength is stored and supplied to the workstation.

A. Initialization and Set-Up Module 50 (FIG. 3)

The initialization and set up module 50, analysis module 52 and displayprocesses 54 of FIG. 2 are part of an application which is launched whenthe user clicks on an icon associated with the application on thedesktop of the workstation display. The initialization and set-up module50 is invoked when the application is launched, indicated at 100 in FIG.3. The initialization and set-up module performs initial tasks that donot require user involvement. The details of the module are notparticularly pertinent to the present invention and therefore manydetails are omitted for the sake of brevity. Basically, with referenceto FIG. 3, the module 50 includes a step 102 which loads run-timelibraries 60 (FIG. 2) which are stored in memory, and which may includeimage processing subroutines and other code modules ancillary to theoperation of the system. At step 104, the module 104 loads configurationfiles, which may contain data pertinent to the particular imaging systembeing used, data pertinent to the specimen being analyzed, and otherconfiguration files. At step 106, the library of reference spectra data(FIG. 2, 64) for the known fluorophores is loaded. At step 108, ainitial screen is presented on the display 44 to the user that allowsthe user to interact with the application, and take initial steps toview the quantitative data, such as select a specimen image set forprocessing, e.g., using a drop-down menu or other tool, select colorsfor individual fluorophores, identify fluorophores which were applied tothe sample, view images of the specimen, view quantitative data, etc.

B Analysis Module 52 (FIG. 4)

The analysis module 52 of FIG. 2 will be described in greater detail inconjunction with the flow chart of FIG. 4 and FIGS. 5, 6 and 8. FIG. 4shows a sequence of individual sub-routines or steps (processinginstructions) which are performed in the module 52 in order to extractquantitative data from the images of the specimen for display on theworkstation.

At step 110, the user is presented with a screen on the workstation bywhich they identify the fluorophores which were applied to the specimen,e.g., by checking a box or by means of selection from a drop-down list.An example is shown in FIG. 8. FIG. 8 shows a display 202 presented onthe workstation display which shows a list 204 of quantum dots or otherorganic fluorophores, and the user checks the box next to the name ofthe fluorophore to indicate that it was applied to the specimen. Theuser checks the box next to autofluorescence to indicate that the userwishes to analyze the sample using a reference spectrum forautofluorescence appropriate for the tissue type being studied.Autofluorescence refers to naturally occurring fluorescence frommolecules present in the sample. Analysis of the specimen to extractautofluorescence data makes use of a separate file containingautofluorescence spectra. The spectra in this file are computed byseparately examining a sample that contains no external (added)fluorophores, and optimizing one or more reference spectra to optimallyrepresent the spectral information collected from this sample.Subsequently, autofluorescence is treated in a manner identical to otherfluorophores in the system, i.e., the fluorophores added to thespecimen.

At step 112, the user selects the labels and colors to use for theindividual fluorophores present in the specimen for display purposes.With reference to FIG. 8, the user is provided with a tool 206 on thedisplay 202 by which the user can select a color for each of thefluorescence types present in the sample for use in display in an imageof the specimen. For example, the user checks on the box “set color”next to the fluorophore and toggles through a sequence of colors toapply to the selected fluorophore. For example for autofluorescence theuser can select blue, for the 605 nm quantum dot fluorophore “Qdot 605”the user can select red, and for the 655 quantum dot fluorophore “Qdot655” the use can select a third color, e.g. yellow.

At step 114, with reference to FIGS. 2 and 4, the module 52 loads theimage cube 70, and the list 72 of wavelengths and the list 74 ofexposure times, described above in the context of FIG. 5.

At step 116, an optional exposure compensation operation is performed onthe M images in order to normalize the response over the range ofwavelengths. It is not unusual for different fluorophores to producefluorescent signals of significantly different peak intensity. Forexample, “QDot 655” produces a much more intense signal than “QDot 525”when excited using the same light source, resulting in an image cubethat is dominated by the “QDot 655” spectrum. To compensate for thissituation and achieve optimal signals from both fluorophores, theauto-expose feature common on camera platforms can be used toautomatically choose the duration for which the camera is exposed to thesample at each wavelength. This results in signal output levels that aresimilar across all wavelengths, but necessitates that a correction beapplied during the quantitative analysis because the fluorescent signalintensity is a function of the exposure time, with longer exposure timesresulting in higher signal intensity. One model that describes thedependence of signal intensity on exposure time is the linear model inwhich intensity scales linearly with respect to exposure time, asdescribed in Y. Garini, A. Gil, I. Bar-Am, D. Cabib, and N. Katzir,Signal to Noise Analysis of Multiple Color Fluorescence ImagingMicroscopy, Cytometry vol. 35 pp. 214-226 (1999). Pixels with anintensity signal equal to the maximum signal produced by the camera maynot behave according to this model and should be either ignored ortreated separately. The details of this treatment are discussed in the‘Constraints’ portion of the Further examples and implementation detailssection.

At step 118, the process 52 performs additional image pre-processingfunctions, including subtraction of background signals (which may not beuniform across all the M images and which may be device-dependent), andapplication of other corrections for imperfections or noise in thespectral filters, the microscope or camera optics, variation in incidentlight output, and camera response. These details are not consideredparticularly pertinent and so a further discussion is omitted for thesake of brevity.

At step 120, the module 52 performs an interpolation of the fluorophorereference spectra (64 in FIG. 2) to the M sampled wavelengths in theimage cube. This operation results in a matrix of illumination intensityat each wavelength, for each of the fluorophores, matrix I in equation(1) below. Each column in the matrix represents the spectral intensityvalue for one fluorophore, at the M wavelengths. Thus, each column has Mrows. There are N columns in the matrix, one per fluorophore. Theinterpolation algorithm also corrects the reference spectra for exposuretime, just like in the exposure compensation operation at step 116.

At step 122, the analysis module 52 performs a spectral unmixing processon the M images in the image cube. A variety of spectral unmixingprocesses are known in the art and considered to be suitable for thiscalculation. In one method, this spectral unmixing process multiplies aMoore-Penrose pseudo-inverse of the matrix I (indicated by [I]⁻¹ inequation (1)) by a vector of the total fluorescence intensity at each ofthe M wavelengths (vector S in equation (1)) to calculate a vector ofconcentration coefficients C₁ . . . C_(N). Methods for calculation of aMoore-Penrose pseudo inverse of an n×m matrix are known in the art. Thisoperation is represented in equation (1) as follows:

$\begin{matrix}{\begin{bmatrix}c_{1} \\c_{2} \\\vdots \\c_{N}\end{bmatrix} = {\begin{bmatrix}{I_{1}\left( \lambda_{1} \right)} & {I_{2}\left( \lambda_{1} \right)} & \ldots & {I_{N}\left( \lambda_{1} \right)} \\{I_{1}\left( \lambda_{2} \right)} & {I_{2}\left( \lambda_{2} \right)} & \; & \; \\\vdots & \; & \ddots & \; \\{I_{1}\left( \lambda_{M} \right)} & \; & \; & {I_{N}\left( \lambda_{M} \right)}\end{bmatrix}^{- 1} \times \begin{bmatrix}{S\left( \lambda_{1} \right)} \\{S\left( \lambda_{2} \right)} \\\vdots \\{S\left( \lambda_{M} \right)}\end{bmatrix}}} & (1)\end{matrix}$

This operation of equation 1 is performed for each pixel location,yielding a vector of coefficients C₁ . . . C_(N) for each pixel. Thecoefficients C₁ . . . C_(N) are related to the concentrations of the 1 .. . N fluorophores present in the sample at each pixel location. Thecoefficients C₁ . . . C_(N) can be scaled to absolute concentrations, orconsidered representative of relative intensity, or relativeconcentration for the 1 . . . N fluorophores, e.g., on a scale of 0-255in an 8-bit quantization of image intensities.

At step 124, the analysis module performs one or more morphologicalprocessing process to identify biological structures that may be presentin the specimen, such as cells, cell membranes, nuclei, viruses, orother. Such processes basically identify the cells or other structuresby identifying patterns and shapes present in an image of the specimen,e.g., closed curves of a certain size. The image upon which themorphological processing operates may one of the M images, a compositeof two or more of the M images, an image constructed from one or more ofthe coefficients C₁ . . . C_(N), a bright field image of the specimen,or other. In a preferred embodiment, the morphological processing isperformed on an image constructed from the coefficients C₁ . . . C_(N).The morphological processing step was described previously.

At step 126, a quantitative analysis is performed for the biologicalstructures, e.g., cells or nuclei, that are identified by the step 124.Such quantitative analysis was described previously.

At step 128, the resulting quantitative data is stored in the memory 38of the workstation for use by the display module or process 54.

C. Display Module 54 (FIG. 2)

The operation of the display module will be described in conjunctionwith FIGS. 7 and 9-18. Basically, this module generates data for displayof images of the specimen and quantitative data to the user on thedisplay of the workstation. A variety of different tools and methods fordisplay of quantitative data will be described.

FIG. 7 shows an image of the specimen which is displayed in a display200 on the workstation. The image may be of the whole slide or a portionthereof. The image is typically multi-colored, with the fluorophoreresponse from each of the fluorophores given a different color, as aresult of the user interacting with the screen of FIG. 8. The imageshown in FIG. 7 is preferably generated by the coefficients C₁ . . .C_(N) which were calculated in the spectral unmixing process.

FIG. 9 is a second image of a specimen. The image 210 of FIG. 9 isgenerated from one of the coefficients C_(i), along withautofluorescence signal. In this instance, the coefficient C_(i)corresponds to a quantum dot conjugated to an antibody which binds to aprotein on the surface of the cell membrane. Hence, in the image, thefluorescence signal due to coefficient C_(i) appears as loops,corresponding to the shape of the cell membrane. The display processgives the user the opportunity to add in to the image the fluorescencesignal from the other fluorophores which are present in the sample. Whentwo fluorophores are present, one signal may be much stronger than theother. Therefore it is preferable to be able to adjust the fluorophorebalance as an aid to visualization of the data. In FIG. 10, the user ispresented with a display 220 with a portion 222 showing two fluorescencecomponents in the sample, in this example autofluorescence and Qdot 625.The display includes a slider bar 224 by which the user can attenuate orenhance each of the fluorophores. For example, the user has enhanced theQdot 625 signal by a scaling factor of 6, which is shown in the box 226.The user can also use the boxes 228 to reset the color of thefluorophores. When the user clicks on the Close button 230, the settingsare saved and the image of both fluorophores with the new colorsselection and weighting is displayed on the display of the workstation.

Recall from the previous discussion of FIG. 8 that the user assigns acolor to each fluorophore for display purposes, and FIG. 10 shows thatthe user can later change that color. For instance, Qdot 525 might beassigned to yellow which in RGB space is (255,255,0)—i.e., the mixtureof pure red and pure green. In general, fluorophore species i has color(red_(i), green_(i), blue_(i)). Once the source data image cubecontaining the overlapping spectra is separated into individual spectra(coefficients C₁ . . . C_(N)), the intensity for each fluorophore ateach pixel is known (on a scale of 0-255). To generate an image thatshows multiple fluorophores mixed together, the colors are linearlymixed in RGB space. Other equivalent approaches are available formerging together, however linear mixing is used in this implementation.When displaying the image showing the contribution from all thefluorophores (an image constructed from all of the coefficients (C₁ . .. C_(N)), the software takes the amount of red contributed from eachfluorophore and computes the overall intensity- orconcentration-weighted red value for each pixel. This is accomplished bytaking the intensity of each fluorophore i (or in equation below“intens_(i)”) times the red component of that fluorophore red_(i) (or inthe equation below “red_(i)”), summing this operation over allfluorophores.

$\begin{matrix}{{red} = \frac{\sum\limits_{i}\left( {{intens}_{i}*{red}_{i}} \right)}{\sum\limits_{j}{red}_{j}}} & (2)\end{matrix}$A similar calculation is performed for the green and blue contributions.

The interface of FIG. 10 allows the user to artificially enhance onefluorophore over another, and achieves this by adding a multiplier infront of the intensity value. Thus, if the user wants to enhancefluorophore k by a factor of 2 (perhaps because it is being hidden byanother, brighter fluorophore), the term in numerator of the aboveequation when i=k would be (2*intens_(k)*red_(k)). This multiplicativeapproach is simply one implementation of enhancement; other methods arepossible.

FIG. 11 shows an example of a display of quantitative data for thespecimen which is presented on the display of the workstation. Thedisplay 250 includes several histograms 252, 254 and 256, severalscatter plots 260 and 262, and an image of the specimen 266. Thehistogram 252 show the size distribution of all cells in the specimen.The histogram 254 shows the distribution of cells showing positiveautofluorescence signal, sorted by cell size. The histogram 256 showsthe distribution of cells showing positive for Qdot 655, sorted by cellsize. The scatter plot 260 plots points which indicate the relativeintensity (or concentration) of one fluorophore as a function of theintensity (or concentration) of one of the other fluorophores, for cellsor other biological structures which are positive for both fluorophores.For example, in the scatter plot 260, the plot shows the amount of Qdot625 as a function of the amount of autofluorescence, which indicatesthat the cells producing relatively low autofluorescence signals alsohad relatively high signal from the Qdot 625 fluorophore (area 258).

The user is able to select any portion of the histograms 252, 254, 256or scatter plots 260, 262 and conduct further quantitative analysis onthe selected portion of the histogram. For example, the user has drawn abox 257 about a certain population of cells in the histogram C (plot256). The box contains the larger cells in the histogram. The scatterplot 262 (plot E) shows the intensity of Qdot 625 signals as a functionof the autofluorescence signal, for the sub-population of cells selectedin box 257 in the histogram 256. The image 266 shows the specimen, withthe selected cells 264 from the scatter plot 262 highlighted. Forexample, the larger cells indicated by the box 257 in the histogram areshown in a contrasting color (appearing as white in FIG. 11).

The display of FIG. 11 further includes additional statistical outputdata 270, such as data showing the intensity or concentration of each ofthe fluorophores for each of the biological structures, mean, median andstandard deviation statistics, and so forth. The user scrolls down usingthe slider bars 274 and 272 to view all of the statistical data.

FIG. 12 shows another screen display presented on the workstation. Thedisplay includes two images 300 and 302, each of which shows the signalfrom one of the fluorophores present in the sample. In this example,image 300 shows the autofluorescence signal and the image 302 shows theQdot 625 signal. The display also includes spectral data 304 for aportion of the image that is selected by the user. For example, the usercan click on or select a point or region in the image 300 or 302, andthe spectra data 304 shows the relative fluorescence data forautofluorescence, the 625 quantum dot, experimental data 312 and modelsum 314. The model sum is the set of values S_(model)(λ) from equation(3) computed based on the I matrix and computed coefficients C₁ . . .C_(N). The experimental data are the S(λ) values that were used inequation (1) to compute the C coefficients. The region 306 allows theuser to select which of the sets of data to present in the spectrum plot304.

$\begin{matrix}{\begin{bmatrix}{S_{model}\left( \lambda_{1} \right)} \\{S_{model}\left( \lambda_{2} \right)} \\\vdots \\{S_{model}\left( \lambda_{M} \right)}\end{bmatrix} = {\begin{bmatrix}{I_{1}\left( \lambda_{1} \right)} & {I_{2}\left( \lambda_{1} \right)} & \ldots & {I_{N}\left( \lambda_{1} \right)} \\{I_{1}\left( \lambda_{2} \right)} & {I_{2}\left( \lambda_{2} \right)} & \; & \; \\\vdots & \; & \; & \; \\{I_{1}\left( \lambda_{M} \right)} & \; & \; & {I_{N}\left( \lambda_{M} \right)}\end{bmatrix} \times \begin{bmatrix}C_{1} \\C_{2} \\\vdots \\C_{N}\end{bmatrix}}} & (3)\end{matrix}$

FIG. 13 shows another example of the display of quantitative data. Thedisplay shows a histogram 254 showing the distribution of cell sizes forcells positive for autofluorescence, and a histogram 256 showing thedistribution of cell sizes for cells positive for Qdot 625 signal. Thedisplay allows a user to select a region in the histogram 256 showingthe cell size distribution for those cells that have a positive Qdot 625signal. Here, the user has indicated the region by drawing the box 257around a group of cells. This action pops up a dialog box 330 that asksthe user what they would like to do with the selected points: show themhighlighted on the image (332), or duplicate another of the plotsshowing only the cells that have been selected at 257. In FIG. 13, theuser in indicating that they want to re-plot Plot-D from FIG. 11 (theintensity plot) with only the points selected in Plot-C (257 in FIG. 13)included. The result is shown in FIG. 14 as a new plot 340, with thescatter plot showing the data points 342 corresponding to thesubpopulation of cells from 257, showing Qdot 625 intensity as afunction of autofluorescence for cells having a non-zero response toboth fluorescence signals.

Next, the points which were selected are shown as highlighted cells on amerged image 350 as shown in FIG. 15. The cellular objects shown inwhite in FIG. 15 are the cells selected in the histogram 256 by the box257 (FIG. 14). The image of FIG. 15 is constructed from theautofluorescence signal and the coefficient C_(i) corresponding to the625 nm quantum dot.

Additionally, the user can note where the interesting cells are on themerged image of FIG. 15, by drawing a region of interest (ROI) 352 onthe merged image window with the mouse. The user is able to inspectspectral information or other quantitative data for that region 352 in avisual results window, shown in FIG. 16. When you select a region on animage as shown in FIG. 15, the curves in the plot on the bottom of thevisual results window (FIG. 16) are updated. This technique may takeadvantage of data linking and data brushing, described in further detailbelow.

FIG. 16 shows an example of an image 350 having two components from twodifferent fluorophores. In this example there is autofluorescence image300 and the 625 nm Qdot image 302.

The tools menu is expanded in FIG. 16 to show the sub-options.Brightness/Contrast 356 launches a tool that allows the user to varythese quantities in the merged image, and Color Equalizer launches thetool shown in FIG. 10 and described previously to allow the user toenhance one or more fluorophores for display purposes only. FIG. 16 alsoshows another example of the spectra in the region 352 selected in FIG.15, including the autofluorescence spectrum 308, the 625 quantum dotspectrum 310, the experimental data 312 and the model sum 314.

FIG. 17 shows another example of the display of images from thespecimen. In the example of FIG. 17, there are three images, namely anautofluorescence image 370, a 605 quantum dot image constructed from thecoefficient C_(i) for the 605 fluorophore, and a 625 quantum dot imageconstructed from the coefficient C_(j) for the 625 fluorophore. Thespectra 308, 376 and 310 for the fluorophores are shown in the plotbelow the images. The images 370, 372 and 374 can be images of theentire specimen, or images of only a portion of the specimen, e.g., asub-region selected by the user as shown in FIG. 15.

FIG. 18 is another example of a display 400 showing a combination ofscatter plots, histograms, image, and statistical data for a specimen.The scatter plot 402 plots intensity of the 605 nm Qdot as a function ofautofluorescence for cells having a positive signal for both types offluorescence. The autofluorescence and 605 Qdot values are plotted on ascale of 0 to 255 for convenience, and the scatter plot 402 is zoomed into magnify the region containing data. The units for the X and Y axis ofthe scatter plots can be scaled to absolute concentration values usingappropriate scaling factors.

Similarly, the scatter plot 410 shows the distribution of the Qdot 655values as a function of the Qdot 605 values, for cells positive for bothfluorophores. The scaling factors on the X and Y axes 412 and 414 couldbe relative intensity or concentration or absolute intensity orconcentration.

The statistical output 422 presents statistical data for all of theidentified objects in the specimen, including the area or size of theobjects, and the intensity values for each object. Additional statisticson the objects, such as mean, median and standard deviation values canbe presented as well.

FURTHER EXAMPLES AND IMPLEMENTATION DETAILS

Z-Stack Imagine

Optical sectioning is the technique of optically imaging “slices” of athree-dimensional sample by changing the focal plane in the verticaldirection and taking images at each plane. See D. A. Agard, “OpticalSectioning Microscopy: Cellular Architecture in Three Dimensions,”Annual Reviews in Biophysics and Bioengineering, vol. 13, pp. 191-219,1984. S. Joshi and M. I. Miller, “Maximum a Posteriori Estimate withGood's Roughness for Three-Dimensional Optical-Sectioning Microscopy,”Journal of Optical Society of America, vol. 10, no. 5, pp. 1078-1085,1993.

The camera of FIG. 1 can use this technique to operate at multipledepths of field (i.e., focus settings in the Z direction, into thetissue sample), and at each depth of field the M images are obtained.The resulting data set includes an image cube at each depth. From thisdata set, three dimensional quantitative data from the specimen isobtained. When the user is presented with the quantitative data, theuser is given an option to select the depth of field they wish to viewand analyze. Additional, quantitative data is obtained for a threedimensional volume of the tissue section. Such quantitative data ispresented to the user.

The separate 2-dimensional images can be registered mathematically intoa 3-dimensional representation of the cells and tissues. One additionalapplication of optical sectioning is to improve resolution of the2-dimensional images. See Enhanced Resolution from Three-dimensionalImaging: W. A. Carrington, R. M. Lynch, E. D. Moore, G. Isenberg, K. E.Fogarty, and F. S. Fay, “Superresolution Three-Dimensional Images ofFluorescence in Cells with minimal Light Exposure,” Science, vol. 268,pp. 1483-1487, Jun. 9, 1995. The concepts of data linking/brushing havebeen extended to 3-dimensional representations. See Pak Chung Wong, R.D. Bergeron, “Brushing techniques for exploring volume datasets,” vis,p. 429, Eighth IEEE Visualization 1997 (VIS'97), 1997. This inventionfurther extends them to include 3-dimension spectral images in thedomain of fluorescence and concentration.

Constraints

The image analysis software uses constraint algorithms to handle errorsin the data. For example, assume that quantitative analysis module isanalyzing a pixel for presence of fluorophores X, Y, and Z. If themodule encounters a pixel that has a negative value Y (a non-physicalsituation), the module re-analyzes that pixel for fluorophores X & Zonly, and uses the new results for X and Z, and sets Y to zero. Also, ifany fluorophore results in a concentration that exceeds the maximum thatcan be detected by the camera, the algorithm sets it to the maximumallowable value. Other approaches to handling anomalous results, such asomitting such pixels from the analysis, are possible, and the algorithmdescribed above is one possible embodiment.

Data-Linking/Brushing

Multidimensional data linking and data brushing are well-accepted meansfor interacting with high-dimensional data. See Interactive DataExploration with Multiple Views (Data Linking and Data Brushing):Sigmar-Olaf Tergan (Editor), Tanja Keller (Editor), Knowledge andInformation Visualization: Searching for Synergies (Lecture Notes inComputer Science) Springer-Verlag, Berlin, 1998. The analysis module ofthis disclosure extends the concepts of data linking and data brushingto the case where one view of the data is in the form of images of cellsin culture or tissue. The image can be either a rendering of the rawfluorescence, or a synthetic image, e.g. images generated from theconcentration coefficients as explained above.

Intelligent Whole Slide Imaging

In systems used in current practice, imaging of biological specimens istypically performed on multiple samples arranged on a slide. Tissuemicroarrays, for example, are paraffin blocks containing as many as 1000tissue samples arranged on a slide in a rectangular fashion. Slides areprepared, using manual operations or automated devices, by taking thinslices of the paraffin material and mounting each slice on a slide. SeeBattifora, H. The multitumor (sausage) tissue block: novel method forimmunohistochemical antibody testing. Lab Invest 1986, 55:244-248;Battifora, H. et al., The checkerboard tissue block. An improvedmultitissue control block. Lab Invest 1998, 63:722-724; Kononen J, etal., Tissue microarrays for high-throughput molecular profiling of tumorspecimens. Nat Med 1998, 4:844-847.

Image acquisition is typically done interactively. The pathologist ortechnician images a whole slide at low resolution and selects regionscontaining tissue samples to view at higher resolution. The selectedregions of interest are then imaged at a higher resolution(magnification), for example if the pathologist wishes to record imagescontaining specific cell types (classified by e.g. tissue type or normalvs. cancer) at high resolution for later processing or review by aspecialist.

In this approach, only small, manually-selected regions are imaged athigh resolution, and detailed analyses of specific regions of interestare typically done at a later time. Because of the large number ofsamples on a slide, and because this process is user intensive andtime-consuming, some samples may not be imaged at the same time, and thesample may have degraded during the delay. Further, the experience andability of the pathologist or researcher may affect which regions areselected, resulting in irreproducible and possibly erroneous results.

One solution to these problems is to record a single image at highresolution. For example, DMetrix, Inc. has developed a scanning systemutilizing an array of microscopes and cameras (see e.g. U.S. Patentapplication publication 2004/0101210 “Miniaturized microscope arraydigital slide scanner”). This approach suffers from two drawbacks. Thefirst is that the data storage requirements for whole slide imaging athigh resolution are prohibitive for many applications. For example, theDMetrix system records a single 12 gigabit grayscale image. While thistechnique could be used with the methods of this invention, if 100 ormore images were to be collected at different wavelengths, it wouldresult in an image cube of 1.2 terabytes or more for a single slide, andhundreds of terabytes for the three-dimensional (Z-stack, multiple depthof field) imaging applications mentioned elsewhere in this document. Thesecond drawback with the existing approach is that standard camerascannot take high resolution images of a whole slide at once because theylack sufficient field of view and the imaging arrays in the cameras arenot large enough.

One method of approaching the goal of an intelligent approach to wholeslide imaging is to generate a low resolution image of the entire slide(or possibly a low resolution image cube of the entire slide) and thenuse the automated image segmentation and classification features of themorphological processing processes to identify important regions on aslide (e.g., biological structures with a high signal for one or morefluorophores) and then collect, preferably automatically, ahigh-resolution image cube of only these important regions on a slide.In one embodiment, the high resolution image cube of the importantregions of the slide is collected shortly after the low resolution imageis obtained, e.g., a few minutes later, after the morphologicalprocessing steps have been performed and the important areas of theslide identified. However, quantum dot fluorophores are less subject todegradation then organic fluorophores and in some embodiments the highresolution image cube can be obtained later on.

The procedure followed in this automated approach includes the followingsteps.

-   -   1) Acquire one or more low-resolution images, or alternatively        an image cube, of the entire slide.    -   2) Using automated image segmentation and classification        algorithms (i.e., morphological processing processes as        described above), areas of the slide which contain regions of        interest such as tissue spots are identified from the one or        more low resolution image, the image cube or an image derived        from the image cube (e.g., an image constructed from        concentration coefficients). Such locations are flagged. The        locations of such tissue spots could be either referenced to        pixel locations in the low resolution image or XY coordinates of        the motion stage that moves the slide relative to the camera and        microscope optics during acquisition of the low resolution        image.    -   3) Acquire higher resolution images of the tissue spots at        multiple wavelengths, basically acquiring an image cube of the        important areas of the slide. The location coordinates from        step 2) are used to position the correct portions of the slide        in the field of view of the camera microscope. The microscope        has a higher magnification objective lens in place for higher        resolution imaging. While the higher resolution image of the        tissue spots are obtained, the camera system records at the same        time metadata summarizing each image, e.g. a slide identifier, a        tissue sample identifier, image magnification and the image        location (or slide location).    -   4) If camera limitations, storage requirements or other        constraints prevent high resolution acquisition of entire tissue        spots, the morphological processing further processes the tissue        spots to identify smaller regions of interest within the tissue        spots using automated image segmentation and classification        algorithms. Such smaller regions of interest could be regions        containing cells, cellular components, genes, DNA fragments,        messenger RNA entities, viruses, or whatever other structures        are of interest in the given assay. Alternatively, such smaller        regions could be only those regions where one or more        fluorophore signal is present.    -   5) Acquire and record high-resolution spectral images (image        cubes) for each region of interest.

The coefficients C₁ . . . C_(N) are calculated for each pixel imagingthe one or more regions of interest or tissue spots in the highresolution image cube. A quantitative analysis of the regions ofinterest or tissue spots is performed as explained previously, includingcalculating fluorophore concentrations for biological structures in thetissue spots or regions of interest from the coefficients C₁ . . .C_(N). The quantitative analysis of the regions or interest or tissuespots proceeds as described above. The display of the results of thequantitative analysis results and images of the regions of interest andtissue spots proceeds using the examples described previously.

The process of automated selection of regions of interest (steps 3-5)can be repeated an arbitrary number of times, with multiple intermediateresolutions sampled before the reaching the required resolution. Thisrequires that the image analysis software system have control of thefield of view and magnification of the microscope, and control of thedigital camera as well. Many commercially available setups provideprogramming interfaces that permit this type of software control of themicroscope and camera. This approach provides for completely automatedanalysis of a whole slide, as well as automated imaging of a whole slidefor later interactive viewing and data exploration.

A similar approach can be used for other types of supports forbiological specimens besides slides, such as, for example, multiwellplates containing cultured cells.

Equivalent Imaging Methods

The data representing the image cube (pixel signal level for rows,columns and at M different wavelengths) can be obtained in a variety ofdifferent orders. For example, we have described generating atwo-dimensional image of the specimen at M wavelengths. Alternatively,one could use a camera such as shown in U.S. Pat. No. 5,926,283 whichcaptures data at multiple wavelengths for one row/column of an image andthen collects the data for the other rows/columns. The end result isstill a cube having rows/columns/wavelengths, with the informationhaving been collected and/o stored in a different order.

As another example, the images making up the image cube could becaptured with a so-called line-scan type digital imager in which thepixels of the camera are arranged in a 1×N linear array of pixels. Inthis type of imager, a row of image data is obtained and then relativemovement between the imager and the slide occurs, then a second row ofimage data is obtained, and so forth, until the entire slide is imaged.To obtain the image cube, rows of image data obtained at one wavelengthare obtained sequentially to image the two-dimensional entire specimenor slide at the first wavelength. Then, a different spectral filter isplaced in front of the camera and rows of image data are obtainedsequentially at a second wavelength. A different filter is placed infront of the camera and rows of image data are obtained sequentially ata third wavelength. The process continues until images of the slide atall the M wavelengths have been obtained.

Software Product for Generic Workstations

The software described above, including the initialization and set-upmodule, analysis module, and the display module, can be loaded on a diskor other machine readable medium and provided as a stand alone productfor commercially available workstations in order to upgrade theworkstation to function as described herein.

In one embodiment, the instructions include a set of instructions for:(a) determining from a set of M images the coefficients C₁ . . . C_(N)for each pixel described above; (b) morphologically processing an imageof the specimen to identify cells or cellular components in thespecimen, (c) conducting a quantitative analysis of the specimenincluding calculating quantum dot concentrations for the cells orcellular components identified in step (b) from the coefficients C₁ . .. C_(N); and (d) generating data for display of the results of thequantitative analysis process (c) on a display associated with theprocessing unit.

Additionally, the software including the initialization and set-upmodule, analysis module, and the display module, can be loaded on anetwork server and executed by a processing unit in the network server.In this case, the user interacts with the software via a clientapplication running on separate computing platform, e.g., a personalcomputer or workstation, which is coupled to the network that includesthe network server. For example, the software may include a Webinterface to allow a remote client to access and view displays of thequantitative data and images of the specimen, and interact with it asdescribed above, but the software processes for calculation thecoefficients, morphologically processing an image of the specimen,conducting the quantitative analysis and generating display data areexecuted in the network server.

Quantum Dots

As used in the claims the term “quantum dot” is intended to be readbroadly to cover luminescent semi-conductor nanocrystals generally,including CdSe nanoparticles as well as CdTe or other luminescentsemi-conductor nanoparticles. Such particles may take any geometricform, including spherical, rod, wires, or other. Gold particles may alsobe used.

All references and literature cited above are specifically incorporatedby reference herein.

While a number of exemplary aspects and embodiments have been discussedabove, those of skill in the art will recognize certain modifications,permutations, additions and sub-combinations thereof as being present inthe disclosure. It is therefore intended that the following appendedclaims and claims hereafter introduced are interpreted to include allsuch modifications, permutations, additions and sub-combinations as arewithin their true spirit and scope.

1. A method of whole slide imaging of a biological specimen contained ona microscope slide, the specimen having 1 . . . N fluorophores presentin the specimen, at least one of which is a quantum dot, comprising thesteps of: a) obtaining one or more images of the whole slide of thespecimen with a digital camera at a low resolution; b) morphologicallyprocessing the one or more images or an image derived therefrom andautomatically identifying one or more regions of interest containingbiological structures present in the specimen; c) automaticallyobtaining an image cube of the one or more regions of interest with thecamera and a set of M spectral filters at a high resolution, the imagecube comprising image data for the regions of interest at M differentwavelengths, where M is an integer greater than 2; d) determining, fromthe image cube and reference data associated with the N fluorophores,coefficients C₁ . . . C_(N) for each pixel imaging the one or moreregions of interest, wherein the coefficients C₁ . . . C_(N) are relatedto the concentrations of the 1 . . . N fluorophores present in thespecimen imaged by each pixel; e) conducting a quantitative analysis ofthe regions of interest including calculating fluorophore concentrationsfor the biological structures from the coefficients C₁ . . . C_(N); andf) generating display data of the results of the quantitative analysisfor display on a workstation.
 2. The method of claim 1, wherein thebiological structures comprises at least one of: cells, cellularcomponents, genes, DNA fragments, messenger RNA entities, and viruses.3. The method of claim 1, further comprising the step of obtaining animage cube of the one or more regions of interest at more than one depthof field.
 4. The method of claim 1, wherein the method is performed by anetwork server and the method further comprises the step of transmittingthe display data to a remotely located workstation over a computernetwork.
 5. The method of claim 1, wherein the specimen is stained withone or more quantum dots.
 6. The method of claim 5 wherein the specimenis stained with both an organic fluorophore and one or more quantumdots.